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SYSTEM AND METHODS FOR REMOTE CONTROL OF MULTIPLE DISPLAYS 

AND DEVICES 

Inventors: 
Jonathan Foote 

Qiong Liu 
Matthew Cooper 
John Boreczky 

FIELD OF THE INVENTION 
[0001] The present invention relates to the control of display equipment. 

BACKGROUND 

[0002] Remote control devices are often used for controlHng electronic equipment such 

as computers, video cameras, Digital Video Disks (DVDs), video tapes, cable broadcast audio 
and video, video projectors, front panel displays, front and back room speakers, user laptops, 
Personal Digital Assistants (PDAs), or other personal display devices. Typically, each electronic 
device has its own remote control. 

[0003] Programmable Universal remote controls have been designed to control multiple 

electronic devices. Programmable universal remote controls typically do not allow the electronic 
devices to interoperate. 

SUMMARY 

[0004] Embodiments of the present invention comprise ways of routing media data and 

controUing media equipment with a handheld controller. 

[0005] In one embodiment of the present invention, a select signal is transmitted to a first 

control device associated with a source device using a remote control. A paste signal is 
transmitted to a second control device associated with a sink device using the remote control, hi 
response to the transmitting, media data is transferred from the source device to the sink device. 
[0006] One embodiment of the present invention is a system including a remote control. 

Control devices are associated with source devices and sink devices adapted to receive select and 
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paste signals from the remote control. The system includes logic adapted to transfer media data 
from one of the source devices to one of the sink devices in response to the select and paste 
signals. 

[0007] In one embodiment of the present invention, a handheld controller is pointed at a 

source device to select media data. The handheld controller is adapted to produce controller 
position and orientation information sufficient to determine the source device. The handheld 
controller can be pointed at a sink device to paste the media data. The handheld controller is 
adapted to produce controller position and orientation information sufficient to determine the 
sink device. In response to the selecting and pasting, the media data is transferred from the 
source device to the sink device. 

[0008] One embodiment of the present invention is a system including a handheld 

controller to produce controller position and orientation information for selecting source and sink 
devices; and logic to transfer media data from a selected source device to a selected sink device 
in response to the selecting of the source and sink device with the handheld computer. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0009] Figure lA is a diagram that illustrates a network system in one embodiment of the 

present invention. 

[0010] Figure IB is a diagram that illustrates a network embodiment of an alternate 

embodiment of the present invention. 

[0011] Figure 2 is a diagram that illustrates the use of multiple control devices in a 

combined display. 

[0012] Figure 3 is a diagram that shows an adjustable remote. 

DETAILED DESCRIPTION 
[0013] One embodiment of the present invention is an intuitive way to route media 

streams and control media equipment using a directional handheld controller. In one 
embodiment, the handheld controller is used to move media files between a source device and a 
sink device. 

[0014] A source device is any device that can generate media data. Media data includes 

(Si 

video, audio still images, a sequence of still images, presentation media such as PowerPoint , 
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scanned or digital documents, a sequence of mouse movements and/or keystrokes, digital "ink" 
from a pen-based computer or electronic whiteboard, steaming internet media, or web pages or a 
sequence thereof. In one embodiment, the media data is a media stream. Possible source devices 
include a Video Cassette Recorder (VCR) or DVD player, broadcast, satellite, or other 
audio/video receivers; streaming internet media decoders; fax or document scanners or digital 
cameras; PDAs or electronic whiteboards; computer systems running PowerPoint®, web 
browsers, internet meeting applications video cameras, microphones, or telephones in locations 
such as a remote conference room. A "null source" can be used to blank displays and to mute 
audio loudspeakers. 

[0015] A sink device is any device that can receive, display, route, broadcast, print, or 

record a media stream. Examples of sink devices include video projectors, Cathode Ray Tubes 
(CRTs) or flat panel displays; laptop or PDA screens; amplified speakers, speakerphones or 
telephones (for audio streams); printers or fax machines; video recorders; and computer systems 
running any number of programs, for example internet meeting applications, electronic 
whiteboard displays, broadcast or CCTV transmission or streaming medi'a encoders for internet 
transmission. 

[0016] In one embodiment, a user connects a source to a sink device by means of a 

handheld controller. Source and sink devices can have one or more associated control devices. 
The user points the handheld controller directly at the indicated control device to control it. In 
one embodiment, a directional controller ensures that only one device receives a control 
command at any one time. In one embodiment, a media data from a source device is selected 
then transferred to an indicated sink device. The "cut" and "copy" functions are both examples 
of the select function. In one example, a "select" button is pressed while pointing at the control 
device associated with a source. A user can then point at the destination control device and by 
pressing the "paste" button direct the media data to that destination. For example, to move an 
image across multiple displays in a conference room, the user could point at the front display and 
press the "select" button. Pointing at a side display and pressing the "paste" button can cause the 
image to be displayed on the side display as well. The front display can then be used to other 
purposes, for example displaying the next image in a presentation. The same source can be 
"pasted" to additional devices, so that copies of the stream can be routed elsewhere, for example 
a display in a remote location for teleconferencing. An optional "blank" button serves to blank 
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unused screens (and/or mute the associated audio) by "pasting" the null source. The "select" 
function can be a "cut" function that includes a "blank" function. These functions give the user 
sophisticated switching and routing control using intuitive point-and-click operations. 
[0017] Figure lA illustrates one embodiment of the system of the present invention. In 

this embodiment, remote control 100 can be used to point to a control device 102a associated 
with a media source device 103. Later the remote control 100 can be pointed to a control device 
102b associated with a media sink device 104. In one embodiment, the remote control 100 
transmits a select signal to a first control device associated with a source device. The select 
signal can be a coded signal that indicates to the control device that media data from the 
associated media source is selected. A paste signal can be transmitted to a control device such as 
a control device 102b associated with a media sink device. 

[0018] In response to the transmitting of the select and paste signals, a media file can be 

transferred between the source device associated with the first control device and the sink device 
associated with the second control device. The control device associated with a source or sink 
device can be located near the source or sink device. Alternately, the control device can be 
embedded within the source sink device. In one embodiment, the remote control wirelessly 
transmits signals to the control device. The wireless device can be a directional device such as 
an infrared (IR) remote control. 

[0019] In one embodiment, logic 106 is used to transfer media file fi-om one of the source 

devices 103 to one of the media sink devices 104 in response to the selected paste signals sent to 
the control devices 102. 

[0020] In one embodiment, the logic 106 includes a central controller 108. The central 

controller receives indications from the control devices 102 that the control devices have 
received a select signal from the remote control 100. The central computer control 108 can 
buffer each media data selection from a source device until a sink device is selected using the 
remote control 100. Once the sink device is selected, the central control 108 can check to 
determine whether the media data is compatible with the destination sink device. If not, the 
central controller 108 can check whether the media data can be converted into a format that is 
compatible with the destination sink device. The source device can be controlled by the central 
controller 108 to output the media data. The central controller can be a computer, dedicated 
processor or other device. This media data can be sent to a switching element, such as a 
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switching matrix 110 which directs the media data to the destination sink device. In some cases, 
the media data can be sent to the media and format converters 112 before being sent to the 
switching matrix. The media and format converters 112 can convert at least some formats media 
data to a format compatible with the destination sink device. For example, a video signal can be 
converted to an audio signal for playing over speakers. Li one embodiment, the central 
controller 108 send media data to the media and format converter 112 to convert the format. The 
switching matrix 110 can be any type of switch including fiill or partial interconnects. The 
central control 108 can store multiple selected media data and source IDS in a select buffer. The 
media data can be buffered on a stack, such as a First-in First-out (FIFO) or Last-in First-out 
(LIFO) stack until the media data is pasted to a sink device. 

[0021] The switching matrix 110 can be configured by the central control 108 to send the 

media data to the correct destination from the correct source device to the correct destination and 
sink device. If the source device and sink device are incompatible or not connectable, the user 
can be alerted by some feedback. The media and format converters 112 can be placed in any 
part of the network. The switching matrix 110 can be implemented as a matrix of smaller 
switches and need not allow all possible routing paths. 

[0022] In one embodiment, the switching matrix can be the Intemet or other network. 

The routing of the media data can be done over the network. The central controller in this case 
need not be centrally located. Logic for the device can connect with the network, such as the 
Intemet to transfer the media data. 

[0023] The system of one embodiment can support a number of powerful media routing 

and conversion operations. 

[0024] Virtual and physical source: In addition to source buttons on the controller, 

isolated IR receivers can serve as sources for selection commands. If a device is physically 
present, for example a DVD player, an IR receiver can be attached to the device. Any "select" 
command directed at this device can store the associated source ID (in this example, the DVD). 
If the source is not physically present, for example a satellite dish, the control device can be 
placed on a wall or other convenient location, perhaps with a sign or icon indicating the name of 
type of the source, and/or a miniature monitor that displays the source media. 
[0025] Virtual Sinks: If a sink device is not physically present in the room, it may still be 

controlled using an associated control device. Examples include a teleconferencing remote 
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display at another location. Small or additional displays can be used to monitor the virtual sinks, 
allowing the user to see the media being sent to the remote location. 

[0026] Media Buses: One class of source/sink is a "media bus." This is a "virtual pipe" 

or media connection through which any media stream can be routed. For example, a bus can be 
established between locations during a teleconference in such a way that any stream pasted to 
that bus would serve as a media source at the remote site, or could be displayed on a dedicated 
sink. 

[0027] Web Virtual Sinks: A useful class of virtual sinks include a web servers that can 

push the indicated media. For example, video can be routed to a streaming media server that will 
broadcast the selected source over the Web. For slide or computer presentations, the virtual sink 
could push still images onto a given web page. This can allow, for example, a local participant 
with a wireless laptop or PDA to access each new image. Additionally, sink devices can include 
a streaming media converter or broadcast station so that media can be accessed from multiple 
remote locations. This functionality could similarly extend to conventional broadcast 
applications in a TV or radio studio. 

[0028] Automatic Media Conversion: Not all sources and sinks are compatible, because 

of resolution, bandwidth, and other factors. Where possible, the server/switcher matrix can 
convert between formats. For example, still frames can be extracted from a video stream and 
pushed to low-bandwidth viewless clients. In an extreme example, a video stream can be 
"pasted" to a printer using automatic summarization system that extracts still images and 
composes them into a printable summary. 

[0029] "Snapshot" mode: Li some cases, it is desirable to capture, store and move only a 

still frame or image (as opposed to a time-based media stream). In this case, a separate control 
button would function similar to the "select" button, but in fact capture a still "snapshot" image. 
This might be particularly useful in a lecture using presentation graphics, as the currently 
displayed graphic could then be pasted to auxihary displays, to user devices (such as laptops for 
further use or annotation), or to a printer or other image-based device. In some cases, the 
lecturer is not the only one with the ability to control the system; given specific controller codes, 
an audience member could "select" the current graphic and "paste" to their personal laptop. 
Given a handheld device with sufficient capability, such as a color screen, it can be possible to 
select a number of "snapshots" which could be represented as thumbnail images in scrolling 

Attorney Docket No,: FXPL-01059US0 MCF/JPO g Express Mail No.: EV 327622660 US 

JOmalley/FXPL/0 1 059us0.flnal application 



display. To recall a desired snapshot, the user would select the desired thumbnail and paste it to 
any available sink. 

[0030] Specific Device Control: Directional hand-held controllers allow specific devices 

to be controlled. Thus, device-specific commands such as play or rewind can be directed to 
specific devices just by pointing at their associated control device. This is an intuitive way to 
control multiple devices fi"om one controller, as shown in Figure 2. Here, five control devices 
are used to control four displays arranged as a "video wall." The control devices at the comers 
control each individual display: control device A controls display 1, and so forth. The central 
control device E controls all four displays as a unit, such that each display shows one quadrant of 
the source stream. Thus to paste a stream only to display 2 at the top left, the user would point 
the remote at control device B. For a larger display, the user would paste the source stream to 
the central control device E, which would result in that stream being shown on all four displays 
configured as one large display. Given the spatial controller described below, on controller 
would serve as a mouse pointer for multiple computer systems with multiple displays. 
[0031] User Feedback: In one embodiment, the system gives users feedback on the 

operations in progress. Because the hand-held controller is directional, immediate visual or other 
feedback on which device is being addressed can be provided. This can be done with visual or 
aural feedback, for example, by lighting a specific indicator lamp on the device being pointed at, 
or by generating a particular sound or vibrating the controller. It is also possible to "highlight" 
the selected system, for example by displaying the selected stream in reverse video or by using 
indicator lights on the selected control device, so that the user knows that stream will be used 
when the "paste" operation is selected. Other examples of "highlighted" sources might include 
adding a distinct border such as a red rectangle, changing the brightness or contrast of a selected 
image, or even changing the room lighting to highlight the selected device. Additional feedback 
can use miniature monitors to display a local or remote stream or device. This is usefiil for both 
sources (for example, a camera at a remote location can be represented with a control device 
having a small monitor that displays the camera image) and sinks (for example, a display at a 
remote location can be represented with a control device having a monitor that mirrors the 
remote display). Feedback on the controller itself may aid control operation, such as a laser 
pointer or tactile feedback that is activated when the controller is pointed at a control device, or 
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an auxiliary display on the controller itself that might show, for example, an image representing 
the currently selected stream. 

[0032] One implementation uses a controller such as an infrared (IR) controller similar to 

a familiar television remote control. Codes can be transmitted by the IR remote to one of the 
possible control devices, which contain IR receivers. Each source and sink has an associated ID 
and control device. Pressing or otherwise indicating the "select" function sends a "select" code 
to the control device being pointed at (which may indicate the selection condition by means of 
lights or other indicators.) This code instructs the system to save the ED for the source associated 
with the indicated control device. Multiple streams may be "selected" in a FIFO stack, such that 
the last stream selected is the first to be pasted. When pointing at the desired sink device and 
sending the "paste" code by releasing the "paste" button, the system connects the source 
(indicated by the stored ID) to the sink display (indicated by the pointed-at device). Thus, the 
signal is routed from one display to another by pointing and clicking. Devices that cannot serve 
as sinks for the selected stream can be made unresponsive to "paste" commands, or can indicate 
that condition by appropriated lights, sounds, or similar feedback. Other buttons generate codes 
that correspond to source IDs (for example the null source). When any particular sink receives a 
source ID code (as opposed to the "selection" code), the server knows to connect that source to 
that sink. Wireless control devices may be used with portable or moveable equipment such as 
PDAs, laptops or equipment on a dolly. In this case, the control device would communicate with 
the system via a wireless link, such as that used for laptop's wireless network connection. 
[0033] An important variable is the beam width of the controller. Too narrow a beam 

will mean that high precision will be needed when pointing the controller; while too wide a beam 
means that potentially more than one device may be selected. To control the beam width, a 
sliding collimator apparatus can be used as shown in Figure 3. For user feedback, indicator 
lights associated with each display could be lit when the source form that display has been 
selected, so the user knows that the source ID is available for a paste. The IR controller can be 
configured to beep or vibrate when a control device is pointed at. Different soimds can be used 
to differentiate between operations such as cut and select. In the preferred embodiment, each 
control device has an indicator light that illuminates when selected. In operation, the user can 
select a command (such as "select" or select snapshot") by depressing a corresponding button the 
handheld controller. This would cause the controller to transmit a corresponding code via an IR 
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LED. Any control device receiving any code can illuminate a "selection" light, however the 
command would not actually be operated upon until the user released the button. Thus, in a 
fashion similar to the familiar menu-selection mouse operation, the user can ensure that the 
desired device had been selected before issuing the command. It can be possible to blink the 
selection lights on multiple devices if they were inadvertently selected, to alert the user to that 
condition, or again to use a tactile or audible alert. 

[0034] A visible laser pointer aligned with the IR beam can be used to assist the user in 

precisely indicating the desired receiver. Alternately, a visible light such as a bright LED can be 
aligned coincident with the IR element in the handheld controller. Thus, anything illuminated by 
the IR source would also be illuminated visible light, making the beam direction and extent 
visible to the user. 

[0035] Figure IB illustrates an alternate embodiment, in which the handheld control 120 

produces a controller position and orientation information such that it can be determined what 
the handheld controller is pointing towards. The handheld controller 120 can interact with logic 
106 to determine the device that the hand held controller is pointing at. The handheld controller 
can be a six-degree-of-freedom (6-DOF) pointer. In this embodiment, the six-degree-of-freedom 
pointer can be used to determine the direction that the handheld controller is pointing at and thus 
determine the media device indicated by the user. 

[0036] The 6-DOF pointer can use a model of the spatial arrangements of media devices, 

typically in a room. A 6-DOF pointer is a device that estimates its spatial location and 
orientation, using triangulation or radio-frequency or other techniques. Examples of 6-DOF 
pointer systems are those made by InterSense and Polhemus. Altematively, a 6-DOF sensor can 
be constructed by using two three-degree-of-freedom sensors attached to a rigid wand or rod. A 
wireless or other communication link can be used between the logic 106 and the 6-DOF 
controller 120. The controller can have sufficient angular resolution to indicate separate devices. 
Each source or sink device can be represented by a two- or three-dimensional region in space, 
which we will call a "control window." For example, an intuitive control window for a large 
display screen would be the rectangle of the screen. Giving the rectangle a small three- 
dimensional depth allows it to be indicated even when the controller is in the plane of the 
window. The system can detect when the controller is pointed at a particular control window by 
geometrically calculating whether the ray coincident with the controller intersects the region 
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enclosed by the control window. Control windows may optionally be surrounded by a "tolerance 
region" such that pointing within the tolerance region would be equivalent to pointing at the 
control window. This would allow the user to select a given control window with less pointing 
accuracy. In the limit, the tolerance region can extend halfway to the nearest control window in 
any particular direction. 

[0037] To connect a stream between source and sinks, the user points the spatial 

controller at the source control window, and the "select" command is indicated using a button or 
other command. This directs the system to store the ID associated with the indicated source 
stream in a "select buffer." Pointing the spatial controller at a sink control window and 
indicating the "paste" command directs the system to route the selected source to the indicated 
sink device. Tactile, audible, and/or visual feedback effects can be used to assist the user in 
selecting the desired command and device. 

[0038] Control windows can be defined in advance, either manually or by using position 

sensors attached to the devices to be controlled. A straightforward application might allow users 
to define control windows using the spatial controller, for example by sweeping out a control 
window boundary while pressing a special controller button, and associating the new window 
with a device using a menu of similar selector. For the user, control windows may be indicated 
by the extent of the control device, by paint, markings or other visual indications, by an active 
projector, or by auditory or tactical feedback. As an example of the latter, the spatial controller 
can be configured to beep or vibrate when an active window is pointed at. Different sounds may 
indicate different conditions, for example an attempt to paste a stream to a device that does not 
support it (such as a source). Location-aware devices, for example a laptop computer equipped 
with a position sensor, can be associated with moveable control windows, such that when the 
device is moved the associated control window moves with it. In this case, the device may be 
controlled by pointing at it regardless of its location. 

[0039] A 6-DOF controller enables a number of applications. For example, the controller 

can be used as a mouse or pen to control a displayed computer screen by manipulating windows 
and buttons, even across multiple computer and displays. The controller can easily be 
configured to act as a mouse or pointer for the display device it is closest to. Similarly, a spatial 
controller could allow a user to control equipment in another room "through" a video projection, 
given sufficient spatial information about the remote room and camera configuration. 
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[0040] By analyzing the location and position of the controller, the system can estimate 

the likely position of the user for video capture, or to display operator assistance, menus or other 
information using an active (steerable) projector. Additionally, the system can detect gestures 
made by moving the controller, and use these as additional commands. In one example, rotating 
the controller in a clockwise direction can direct the system to display the next slide in a 
presentation, while counter clock-wise motion can indicate the reverse. A special control 
window can be associated with a region on a desk or wall, and controller motion in this window 
could be interpreted as pen strokes for display elsewhere. 

[0041] The foregoing description of preferred embodiments of the present invention has 

been provided for the purposes of illustration and description. It is not intended to be exhaustive 
or to limit the invention to the precise forms disclosed. Many embodiments were chosen and 
described in order to best explain the principles of the invention and its practical application, 
thereby enabling others skilled in the art to understand the invention for various embodiments 
and with various modifications that are suited to the particular use contemplated. It is intended 
that the scope of the invention be defined by the claims and their equivalence. 
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